An Implementation of a Multilingual Regular Expression Segmentor for Ordinary and Morphologically Rich Lexical Tokens

Lexical pattern matching and text extraction is an essential component of many Natural Language Processing applications. Following the language hierarchy first conceived by Chomsky, it is commonly accepted that simple phrasal patterns should he categorised under the class of Regular Language (RL). There are 3 operations in RL Union, Concatenation and Kleene Closure which are applied to a finite lexicon. The machinery that recognises RL is the Finite State Machine (FSM). This paper discusses and postulates that the degree with which a class of patterns exercise the aspects of RL operators, is directly proportional to the richness in morphology of lexical tokens.